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ABSTRACT 


Demonstration of required reliability performance levels prior to system fielding 
has remained a challenge for the Army, and in recent years, the success rate of systems 
achieving their stated reliability performance in operational tests has declined. 
Realization of required reliability performance necessitates effective management 
strategies and techniques in order to reduce risks. “Designing-in” reliability up front is 
one of the critical elements to insure adequate system level reliability. It has also been 
ranked as one the top reliability problems by Program Managers. One design technique 
for maximizing inherent reliability is through the use of highly reliable parts, materials, 
and processes. Reliable parts, materials, and processes are the building blocks of the total 
system reliability. They have a significant role in overall program success because it 
contributes to the ability to meet its cost, schedule and performance goals. 

This research concentrates on the impacts of highly reliable parts and materials on 
the overall reliability of a weapon system. To gather these data, the researcher drew 
directly from experiences of part, material, and process professionals from the Army, 
Navy, Air Force, and contractors with whom they interface. Results show that the key to 
success resides in early involvement of part, material, and process experts along with a 
process that facilitates communication and open dialogue. Maximizing inherent 


reliability is the desired end state. 
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I. INTRODUCTION 


A. BACKGROUND DISCUSSION 


The Army Vision, announced in October 1999, is to transition the entire Army 
into a force that is strategically responsive and dominant at every point on the spectrum 
of operations. This equates to a need for high readiness levels for rapid deployment, as 
well as a significantly reduced logistics footprint in the battlespace without jeopardizing 
combat capability. One enabler for achieving this is highly reliable systems. Highly 
reliable systems are force effectiveness multipliers, as the resulting benefits contribute 
towards reduced maintenance times, increased system availability, reduced training and 
manpower, fewer spare parts, and a net reduction in total ownership costs (TOC) that will 


free up scarce funds needed for Army modernization. 


Demonstration of required reliability performance levels prior to system fielding 
has remained a challenge for the Army. According to the Army Test and Evaluation 
Command (ATEC), the success rate for systems in operational testing over a five-year 
period from 1996 to 2000 was only 20 percent. [Ref. 1] The failure of a weapon system 
to meet its predicted reliability levels could have a severe impact on its future logistics 
and life-cycle support costs. According to a recent analysis performed on the Army 
Comanche helicopter, missing predicted reliability goals by just one percent could result 


in an increase in lifecycle O&S costs of over $75 million. [Ref. 2] 


The reliability development process is composed of many elements (e.g., a PM&P 
Program, Materials and Processes, Test and Evaluation, Design, Design Interface, 
utilization, etc.), including the management process of establishing a reliability factor for 
the system. This management process is such a crucial part of the development process 
because, as reliability is allocated among the subsystems, it becomes a design driver. A 
program can have a strong PM&P Program, T&E, etc., but if an unrealistic reliability 
factor is established, the program will not have a reliable system, nor will it have 
achieved its reliability requirement. As a rule, reliability is generally overstated, and the 


impacts on test, support, inventory, personnel, and training can be harmful. [Ref. 3] 


“Designing in” reliability up front is one of the critical elements to ensure 
adequate system level reliability. It has also been ranked as one the top reliability 
problems by Program Managers. [Ref. 1] One design technique for maximizing inherent 
reliability is the use of highly reliable parts, materials, and processes. Reliable parts, 
materials, and processes have a significant role in overall program success because they 
contribute to the ability to meet cost, schedule and performance goals. One of the 
techniques to assure the use of highly reliable parts, materials, and processes is through 


the use of a Parts, Materials, and Processes (PM&P) Program. 


A PM&P Program can be applied in several important ways over the life cycle 
of a system. An effective PM&P Program is key to mitigating risk associated with the 
use and disposition of noncompliant hardware, avoiding costly redesigns, attaining costs 
savings from reduced need for failure analysis, rework, etc. Throughout this thesis, when 
referring to a PM&P Program, the capitalized acronym will be used. When referring to 
the parts, materials, and processes themselves, no capitalization will be used and the 
words will be spelled out. 


B. OBJECTIVES AND PURPOSE OF THE RESEARCH 


Required reliability performance depends upon effective management strategies 
and techniques to address reliability risks over the course of a weapon system’s 
development and fielding. This research concentrates on the impacts of highly reliable 
parts and materials on the overall reliability of a weapon system. In DoD terms, it will 
focus on essential steps that a Program or Logistics Manager can take to maximize 
system component reliability. A large part of the effort focuses on giving the reader an 
understanding of how an acceptable reliability factor is established for the system, the 
importance of achieving high reliability, and the relationship among reliability, logistics 
and life-cycle support costs. The objective is to determine how to best maximize the 
inherent reliabilities of system components through the use of highly reliable parts and 
materials. 


C. RESEARCH QUESTIONS 


The Primary research question is twofold: how is the selection of highly reliable 


parts and materials an essential part of the overall reliability development process, and 
2 


what critical tactics and strategies can Program and Logistics Managers use to maximize 


the inherent reliabilities of system components? 


Subsidiary research questions are: 

1. What technology, tools, and techniques are available to ensure the proper 
selection of reliable parts and materials? 

2. How has acquisition reform changed the way a PM&P Program is managed? 

3. To what extent does commercial industry differ in its approach to part and 
material selection, and can the Military leverage these best practices to 


improve performance in military systems? 


4. How cana Program and Logistics Manager mitigate part and material risks? 


D. SCOPE 


The thesis focuses on the tactics and strategies that Program and Logistics 
Managers can use to ensure the highest possible inherent reliabilities of system 
components. Although the data will originate primarily from Army acquisition 


professionals, lessons learned should be applicable to the other DoD Services. 


The scope is limited to an evaluation of reliability considerations in the system 
design process from several aspects: (1) the technology, tools, and techniques that are 
available to ensure the proper selection and use of highly reliable parts and materials; (2) 
an analysis of how parts and materials are managed under acquisition reform; and (3) the 
extent to which commercial industry differs in its approach towards part and material 
selection and how the DoD can leverage these best practices. The thesis concludes with a 
recommendation for actions or steps a DOD Program and Logistics Manager might take 
to maximize inherent reliability through the use of highly reliable components. 


E. METHODOLOGY 


The methodology used in this thesis research consists primarily of two steps. The 
first step is to demonstrate why a weapon system with a high inherent reliability must be 
designed using highly reliable parts and materials. The importance of selecting an 
acceptable reliability requirement will be established. The connection will be made 


between part and material, reliability, logistics, and life-cycle support costs. Then the 
3 


thesis explains the benefit of “designing in” reliability and how the selection of flawed 
system level reliability affects design at the lower levels. Next, the thesis provides an 
overview of the contemporary reliability environment within the DoD. Acquisition 
reform, current DoD PM&P Program initiatives, commercial best practices, and 
contracting methods for effective part and material selection are evaluated using literature 
reviews and interviews with acquisition professionals. The thesis also addresses program 
management techniques in the same manner. A comprehensive literature review on the 
subject of reliability encompasses material and sources that include, but are not limited 
to: 

1. DoD, Army, Navy, and Air Force publications 

2. Published academic research papers 

3. References, publications, and electronic media available at the Naval 

Postgraduate School (NPS) 
4. World Wide Web Sources (DoD, commercial, academic) 
5. Personal communication with School of Business and Public Policy faculty at 


NPS 


The second step entails interviews with subject matter experts (SMEs) in the 
parts, materials, and processes functional area. Data gathering and analysis were 
conducted through structured interviews. The analysis synthesized feedback and 
searched for collective issues from the various SMEs on managing PM&P Program 
requirements, the coordination that is required, the common issues, reasons why they 
occur, and how these risks can be reduced. To address the concern of data reliability, the 
SME had editorial control. The interview answers were typed and given to the SME for 
review and approval prior to incorporation. 


F. ORGANIZATION OF THE STUDY 
Chapter I. Introduction: identifies the focus and purpose of this thesis as, well as 
the primary and subsidiary research questions. 


Chapter II: Overview and Background: provides a literature review, giving the 


reader a basic understanding of the importance of establishing an acceptable reliability 


factor for the system, achieving high reliability, and the relationship among reliability, 


logistics and life-cycle support. 


Chapter III: Managing a PM&P Program on a Weapon System: presents the data 
collected from interviews with subject matter experts (SMEs) in the parts, materials, and 


processes functional area. 


Chapter IV: PM&P Program Analysis and Lessons Learned: analyzes and 





compiles the PM&P Program technologies and tools, the impact of acquisition 
streamlining, commercial PM&P Programs, and the techniques to mitigate part and 


material risks. 


Chapter V: Conclusions and Recommendations: summarizes the findings of the 





research and answers the research questions. 
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Il. BACKGROUND 


A. INTRODUCTION 


The purpose of this chapter is to provide insight into how a weapon system 
reliability requirement is developed and to explain its relationship to highly reliable parts, 
materials, and processes. The chapter first discusses the current reliability trend in the 
Army. It then describes the PM&P Program as it relates to the acquisition process in 
order to make the analysis presented Chapter IV more understandable. Next, the chapter 
details the critical need to determine the proper reliability requirement for the system. It 
then discusses the importance of achieving high reliability and the relationships among 
reliability, logistics and life-cycle support costs. The benefits of “designing in” reliability 
and how those benefits relate to the PM&P Program are reviewed, along with the impact 
of the system-level reliability requirement on the individual parts, materials, and 
processes. Former and current DoD policies, as well as commercial PM&P Program 
practices, are covered. 


B. THE CURRENT RELIABILITY TREND WITHIN THE ARMY 


According to the Army Test and Evaluation Command (ATEC), the success rate 
for Army systems either in development or operational testing over the five-year period 
from 1996 through 2000 was only 41%. In the same five-year period, the system 
operational test (OT) success rate with respect to reliability was only 20%. [Ref. 4] 
Figure 1 represents operational test events that were used as the basis for demonstrating 
reliability requirements. All acquisition category (ACAT) levels are represented here. 
The types of OT events included: field exercises; IOTs, FOTs, LUTs; and combined 
DT/OT. Points above the diagonal achieved their reliability requirement during testing, 
while those below did not. Most of the systems failed to achieve their reliability 


requirement in OT, and the trend appears to be downward. 
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Figure 1. Army System Reliability Performance [From Ref. 5] 


C. DESCRIPTION OF A PM&P PROGRAM 


There is no universal definition of a Parts, Materials, and Processes (PM&P) 
Program, although individual definitions for parts, materials, and processes are clearly 
defined in DOD specifications (e.g., MIL-STD-1547 and MIL-STD-965). For the 
purposes of this research, a Parts, Materials, and Processes (PM&P) Program is defined 


as: 

A process that deals with individual Parts, Materials, and Processes 

requirements. [Ref. 6] 

In order to supplement the earlier definition of a PM&P Program, a part is further 
defined as: 


One piece, or two or more pieces joined together, which is not 
normally subject to disassembly without destruction or impairment 
of intended design use. [Ref. 7] 


Requirements are defined as: 


Characteristics that individual items must have within a stated 
environment or a set of conditions as derived from the systems 
engineering process. [Ref. 6] 


The conditions or environments mentioned above are based on the ORD-derived 
constraints and performance requirements, as reflected onto the individual part, material, 
or process. Parts, materials, and processes are critical to most acquisitions and are a 
fundamental element in the achievement of designed hardware performance, which 
includes: 

1. Analysis to support the establishment of comprehensive PM&P Program 

characteristics from the derived system engineering constraints and performance 


needs. 


2. Establishment of a cross-functional Plan for efficient and uniform 


implementation of a PM&P Program. 


3. Analysis and documentation to support individual part, material, and process 


needs, consistent with PM&P Program requirements. 


4. Development, maintenance, and control of databases documenting design 
requirements, design baseline, control and use of life-limited (items having a 
finite shelf life), the basis for decisions made, and control of evolving 
requirements, so that the impact of changes can be effectively determined prior to 


implementation. 


5. Development of a closed-loop system to efficiently feed back necessary 
changes derived from system-level performance results and industry data 


interchange. 


6. Development of In-Process or End-of-Line validation processes for known 


good parts, materials, and processes. 


7. Development, maintenance, and control of program-compliant parts, material, 


and process lists. 


A PM&P Program is applied in several important ways over the life cycle of a 
weapon system. For example, during the Concept and Technology Development and 
System Development and Demonstration phases of a weapon system, a PM&P Program 
is applied iteratively as an integral part of system engineering to mitigate risk and 
tradeoffs between system performance, ease of maintenance and manufacture, cost, 
production level, parts and materials availability, and technological advancements. 
Ineffective use of parts, materials, and processes at this early stage of a program may 
cause unnecessary future program costs and delays. [Ref. 6] 


D. ESTABLISHING AN ACCEPTABLE RELIABILITY REQUIREMENT 
1. Origination of the Reliability Requirement 


Reliability has been defined as “the probability that a system will operate within 
the designed specified limits for a particular period of time when used in the manner 
intended”. [Ref. 8] In the Army, the reliability requirement starts with the Combat 
Developer (CBTDEV). The CBTDEV develops the Operational Requirements 
Document (ORD) and, hence, is ultimately responsible for defining the system’s 
reliability requirements. Typically, these are defined in terms of operational availability 


and mission duration needs. 


Reliability requirements, however, are not developed in a vacuum. Developing 
quantitative operational reliability requirements, like all other ORD requirements, is a 
collaborative process between the CBTDEV and the Materiel Developer (MATDEV) 
using Integrated Product Teams (IPTs). IPTs provide a balanced solution between the 
best estimate of what will meet the user’s effectiveness, suitability, and survivability 
needs and what is actually affordable and technically possible within program funding, 
risk, and time constraints. ORD reliability requirements are developed in accordance 
with AR 71-9 [Ref. 9]. Three key elements combine to define overall reliability 


performance requirements: list them briefly. 
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2. Program Manager’s Role in the Reliability Requirements Process 


Once the overall system requirement is established in the ORD, the DoD 
delegates the responsibility for implementing the reliability requirement to the Material 
Developer (MATDEV), who is usually a Program Manager (PM). The PM is responsible 
for the system acquisition, including the reliability requirement. DoD 5000.2-R states 
that the “the PM has the task to ensure that reliability, maintainability, and availability 
activities are established early in the acquisition cycle to assure meeting operational 
requirements and reduced life-cycle ownership cost." [Ref. 10] The Army further 
requires the PM to develop and implement an effective Reliability & Maintainability 
(R&M) program through AR 70-1, which states: 


[An] R&M program will be tailored in scope and content and be designed 
to ensure that the user operational reliability requirements will be met at 
confidence levels established by the user. 


Finally, DA PAM 70-3 guidance covers aspects of R&M Requirements, R&M 
Management, R&M Engineering and Design, R&M Testing, and R&M and Assessment 
Integrated Process Team (IPT) procedures. [Ref. 11] The PM must take a proactive role 
in establishing systems reliability, starting at program initiation and continuing through 
the entire program life cycle. 

3. Contracting for Reliability 

It is the now the PM’s responsibility to insure that reliability objectives are 
translated into quantifiable and verifiable contractual terms. Prior to the advent of 
military specifications and standards reform in 1994, the work requirements for reliability 
engineering were usually described in a Statement of Work (SOW) task that required 
compliance with MIL-STD 785, “Reliability Program for Systems and Equipment 
Development and Production.” [Ref. 12] In February 1996, Gil Decker, the Army 
Acquisition Executive at the time, issued policy on incorporating a performance-based 
approach to Reliability in Requests for Proposals (RFPs). A key change was that no 
“how to” reliability standardization documents were to be used. The policy stated: 


Reliability requirements should be included in RFPs by specifying: 
(1) quantified reliability requirements and allowable uncertainties, (2) 
failure definitions and thresholds, (3) life-cycle usage conditions. [Ref. 13] 
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Decker’s policy was institutionalized in the update to AR 70-1 in January 1998. 
AR 70-1 clarified several points of the AAE memo. 


Reliability parameters expressed by operational users, and ones specified in 
contractual documents, take many different forms. User requirements are generally 
expressed in a variety of forms that include combinations of mission and logistics 
reliability, or they may combine reliability with maintainability in the form of 
availability. Conversion from commonly used operational terms, such as mean-time- 
between-maintenance (MTBM) and mean-time-between-critical-failure (MTBCF), must 
be made to enable translation to parameters that can be specified in contracts and verified 


in testing. 


In order to verify that a contractor will achieve the system reliability requirement, 
the PM will also include contract provisions that ensure access to sufficient information 
for evaluating source data, models, reasonableness of modeling assumptions, methods, 
results, risks, and uncertainties. [Ref. 14] Also, the test program must be structured so 
that the reliability requirements can be demonstrated prior to full-rate production. One of 
the cardinal rules of testing is to verify contractor claims of reliability as early as 
possible. [Ref. 15] The information used to estimate or determine progress toward 
meeting R&M requirements must be based on testing in accordance with the system’s 
expected mission profile and operating environment. Reliability-growth projections or 
other prediction methods are sometimes used in determining that contractual reliability 
requirements have been met. 


4. Reliability Allocation 


After an acceptable requirement has been established for the system, it must be 
allocated among the various subsystems. Reliability allocation involves setting reliability 
objectives for components or subsystems in order to meet a system-level reliability 
objective. Reliability allocation should occur in the initial stages of design. A 
knowledge of technical capability and cost trade-off considerations will help derive 
optimum allocations, but the simplest method for allocating reliability is to distribute the 
objective uniformly among all subsystems. For example, consider a simple system 


composed of two subsystems: subsystem A and subsystem B. Each subsystem is 
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composed of assemblies (assembly A through assembly P). Each assembly is composed 
of components (component | through component 10). The reliability of the system is 
based on the reliability characteristics of the subsystems; the reliabilities of the 
subsystems are based on the reliability characteristics of the assemblies; and so forth. At 
the lowest level (i.e., the component level), reliability is assumed or predicted. [Ref. 16] 
Figure 2 displays reliability block diagrams for this example—created using a sub- 


diagram approach—and parameters for each component. 


RELIABILITY ALLOCATION 


PURPOSE: 
- IN-DEPTH ANALYSIS ENSURING THAT REQUIREMENTS ARE ALLOCATED 
FROM THE SYSTEM LEVEL TO THE REQUIRED SUB-LEVEL 























RELIABILITY = 0.999 PER COMPONENT 


Figure 2. System Reliability Allocation Diagram Example 


5. Causes and Impacts of Establishing a Flawed Requirement 


One of the greatest challenges designing a weapon system is establishing an 
acceptable reliability requirement. Sometimes, established reliability requirements are 
not realistic. Although the PM is supposed to be involved in generating requirements, in 
reality, R&M Managers from the program office are given little voice during this part of 
the process. The checks and balances to ensure a proper requirement were either 


removed as part of acquisition streamlining or are simply not in place. [Ref. 3] 


Program offices must also deal with unrealistic contractor estimates. During 
initial acquisition, the contractor engineers provide reliability estimates that are critical in 
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establishing sparing levels and technical support criteria. Experience has shown that 
contractors, who are under pressure to win and maintain contracts, use optimistic 
reliability predictions. [Ref. 17] The null hypothesis of having the contractor verify 
reliability estimates as accurate computations is seldom enforced. Most weapon systems 
are cost- and schedule-driven and do not incorporate the time or resources necessary to 


perform reliability tests on critical sub-systems. [Ref. 3, Ref.17] 


The impacts of a flawed requirement are two-fold. Not only will the system have 
difficulty meeting the requirement during testing, but the development of a sustainment 
package based upon a flawed reliability factor also will produce sparing shortages, test 
equipment shortages, and decreased readiness. [Ref. 15] 

E. HIGH RELIABILITY AND THE LINK TO LOGISTICS AND LIFE-CYCLE 

SUPPORT COSTS 


The failure of a system to achieve high reliability levels has a severe impact on 
operation and support (O&S) costs and operational availability (Ao). The impact of 
reliability on O&S costs is attributable to the high costs associated with acquiring repair 
and replacement parts. High reliability is fundamental to cost-effective and efficient 
logistics; it means less support and training and fewer inventory costs and personnel. 
According to an analysis of the Army Comanche Program, missing predicted reliability 
goals by just one percent could result in an increase in O&S costs of over $75 million. 
[Ref. 2] Figure 3 demonstrates the huge O&S costs associated with maintaining some 
other Army weapon systems. As the chart shows, the potential impact on lowering O&S 


costs through increasing reliability is enormous. 
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Figure 3. Cost to Support Current Reliability Levels [From Ref. 5] 


The impact of reliability on readiness also can be significant. In the Comanche 
Program, for each five percent below the predicted reliability level for peacetime 
operations, the Ao decreases by one percent. [Ref. 2] Figure 4 demonstrates the impact 
of reliability on Army tank availability. The impact is significant, as 45 percent of the 


tanks were down at one point during the gunnery firing exercise. 
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Figure 4. Battalion’s Tank Fleet Hit 45 Percent Down During A Gunnery [From Ref. 5] 


Reliability also affects troop morale. Because low reliability causes a high failure 
rate, it makes unrealistic demands on maintenance personnel. Repairs and maintenance 
may have to be performed very quickly at any time, day or night, in order to meet 
operational commitments. In such cases, personnel must continue working until the job 
is done, regardless of how much time it takes. An August 1999 GAO survey of active- 
duty personnel showed that a significant portion of aviation maintenance personnel 
worked more than 50 hours a week, and that some worked 70 hours a week. The GAO 
reported that the majority of factors cited by military personnel as sources of 
dissatisfaction and reasons for leaving the military were work-related, such as the lack of 
parts and materials to successfully complete daily job requirements. One example is an 
aircraft wing with a status report showing that six out of 28 aircraft downed for parts had 
not flown for 37 days or more. One of these aircraft had not flown for more than 300 
days and, according to the Maintenance and Material Control Officer, was missing 111 
parts. As a result, the Navy had been unable to use this multimillion-dollar asset for 


almost a year. [Ref. 18] 


When faced with supporting maintenance-intensive weapon systems, technicians 
and logisticians are initiated into a culture of learning to do more with less and 
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performing the mission at all cost. Failure to perform a mission successfully is deemed 
an unacceptable outcome. Rather than fail, technicians (and commands) resort to 
shortcuts, work-arounds, and misrepresentations of readiness to feign the appearance of 
mission readiness. [Ref. 19] To the extent that low reliability causes failures that 
contribute to extensive overtime, morale and retention will be adversely affected, and 
additional costs may be incurred in recruiting and training new personnel 


F. BENEFIT OF “DESIGNING IN” RELIABILITY 


Although the reliability development process consists of many elements (e.g., a 
PM&P Program, Test and Evaluation, Design, Design Interface, utilization, etc.), the 
benefit of “designing in” reliability cannot be understated. “Designing in” reliability 
implies performing up-front analyses during the design phase so that the inherent 
reliability of the system is as high as possible. As with anything else, being proactive 
with reliability early in the lifecycle of a system is more cost-effective than dealing with 
potential schedule delays and unexpected costs of failing a test later, only to have to 
redesign and test yet again until the problem is fixed. It is critical to make reliability part 
of the early design because the decisions made during this period have such a great 
impact on the Life Cycle Cost (LCC). The chance to influence LCC cost reductions 
grows smaller when projects are converted into bricks and mortar, as Figure 5 illustrates. 
Making major changes in LCC after projects are turned over to production is not possible 


because the die has been cast. [Ref. 20] 
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Figure 5. %LCC vs. Life Cycle Span [From Ref. 20] 


In the past, the Army has relied on costly test-analyze-fix (i.e., reliability growth) 
programs in order to "grow" reliability until requirements were met. The decision to 
“design in” reliability depended on the individual program. There are, however, 
significant benefits to “designing in” reliability. Designing in reliability will achieve 
higher system reliability up-front, thus reducing unforeseen and costly failures during 
later phases in a program. Highly reliable components will significantly reduce a 
system's O&S costs and, thus, reduce the expensive process of reliability growth and the 
resulting project delay costs. It is crucial to emphasize, early on, the use of proper design 
tools and activities to “build in” reliability up-front, rather than relying on extensive 


“back end” testing and validation. [Ref. 21] 
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In a briefing on 22 February, 2001, Tom Edwards, Deputy to Commanding 
General Combined Arms Support Command (CASCOM), told the International Society 
of Logistics that reliability can be significantly improved by focusing on two main areas: 
1) designing in reliability up-front, and 2) validating that we have reliable systems and 
components. This highlights the concept that reliability can be significantly improved by 


design early in the development process. 


If a fielded weapon system is to have a high system readiness objective—either 
operational availability or probability of mission success—it must have high reliability. 
It generally is understood that to have a high level of readiness, one must have a reliable 
system. [Ref. 8] For a system to be reliable in the field, it must have good inherent 
design reliability that starts with the original design concept and continues throughout the 
development of the design. To reduce the risk of creating an unreliable system, the 
design should be required to follow certain reliability and maintainability practices and 
guidelines. It is essential that these practices and guidelines begin early in the design 
concept because poor or marginal reliability is often difficult to detect before the design 
effort is completed. The following section presents the benefits of using highly reliable 
parts, materials, and processes to achieve a high inherent reliability. 


G. USING A PM&P PROGRAM TO “DESIGN-IN” RELIABILITY 


Simply stated, highly reliable parts, materials, and processes are the building 
blocks of a highly reliable weapon system. While the system design approach is largely 
responsible for the weapon system’s functional performance, the PM&P Program 
approach is responsible for a large part of the weapon system’s reliability performance. 
Therefore, to insure that the reliability performance is being met, one has to evaluate the 
PM&P Program requirements, their implementation and verification, and the 
requirements’ translation to the lowest part, material and process applied in the system. 

The goal of a PM&P Program is the selection of parts, materials, and processes 
that can withstand the manufacturing environment and reliably perform the needed 
function for the design life of the product in the intended environment. Selection 
involves designing with parts and materials that have known reliability, superior life, 


durability, and corrosion resistance. Application involves using a PM&P Program to 
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ensure that field environment stresses are less than the parts’ and materials’ strengths. 
Individual part, material, and process selection criteria and a PM&P Program should be 
used in the preliminary design to ensure that designers are provided preferred parts 
selection lists before design begins and that they continue to use only approved parts, 
materials, and processes in the system design. Examples of the adverse effects from 
designing without a PM&P Program include unknown failure mechanisms, inappropriate 


corrective actions, inappropriate design rules, low system reliability, etc. [Ref. 6] 


The following case of microcircuits in a tactical missile offers an example of how 
a PM&P Program affects system-level cost and reliability. Suppose that the cost of the 
missile is $100,000, and the missile contains 70 microcircuits, each of which can cause a 
mission-critical failure. If 2000 missiles are produced, then there will be a total of 
140,000 microcircuits. Now further assume that each microcircuit has 0.9989 reliability, 
and the reliability is allocated serially (as in Figure 2). This means that, on average, there 
will be 148 failed missiles and a lost value of $14,800,000. Now assume that there is a 
reliability test that can ensure 100 percent reliability and that this test will be performed 
on each lot of microcircuits at $12,000/lot. The total cost of the test would be $924,000 
(with tests for seven additional lots). The lost value exceeds the test cost by ~16:1 and 
would likely justify the test cost to avoid the losses. This, of course, ignores the 
potentially life-threatening consequences of a tactical missile failure. [Ref. 22] 
H. FORMER AND CURRENT DOD PM&P PROGRAM POLICIES AND 

GUIDANCE 


The PM&P program is founded on DoD 4120.24-M Defense Standardization 
Program Policies and Procedures, which establishes the current policies and procedures 
to achieve the objectives required by 10 U.S.C.2451-2457, DoD Directive 5000.1, and 
DoD 5000.2-R. Under the DoD’s performance-based acquisition policies, it is primarily 
the contractor’s responsibility to recommend the use of materials, parts, components, and 
other items needed to meet performance requirements and satisfy other program 
elements, such as parts management and logistics support. Paragraph C3.2.4 of DoD 


4120.24-M contains the following mandatory requirement: 
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Program offices shall ensure that a_ parts 
management process is used to reduce the proliferation of 
parts and associated documentation and promote the use of 
parts with acceptable performance, quality, and reliability. 
[Ref. 23] 

Prior to acquisition reform, detailed parts control program requirements for use on 
newly designed and/or modified equipment were typically invoked in contracts through 
Task 207 of MIL-STD-785. Task 207 required the establishment of a parts control 
program in accordance with MIL-STD-965. Figure 6 illustrates the relationship of MIL- 
STD-965 to MIL-STD-785 and other parts and material specifications. One of the 
provisions of MIL-STD-965 required contractors to submit parts lists to the Government 
for approval. Another provision was a part selection order of preference, in which 
military specification parts were considered first, and commercial parts could be 


considered only when no military part was available. 
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Figure 6. Reliability Design/Parts Application Specifications Prior to Acquisition 


Reform [From Ref. 24] 


In order to be consistent with acquisition reform initiatives, MIL-STD-965 was 
converted to MIL-HDBK-965 in September 1996. In October 2000, MIL-HDBK-512 
was released, and MIL-HDBK-965 was cancelled. MIL-HDBK-512 now provides the 
current guidance (guidance on/ly— if it is cited as a requirement, the contractor does not 


have to comply) for implementing an effective parts management program on 
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DoD/Industry acquisitions. MIL-HDBK-512 supports the acquisition strategies and 
systems engineering practices of DoD 4120.24-M and DoD 5000.2-R. It also provides 
performance-based parts management process guidance, including the objective to 
enhance the reliability of end-item performance. MIL-HDBK-512 no longer requires 
contractors to submit parts to the Government for approval; nor does it require the 
preference of military specification parts over commercial parts. The management 


elements of MIL-HDBK-512 include: [Ref. 7] 


1. Parts selection baseline. This is a parts selection list or database that should be 





maintained in order to give to designers and subcontractors the list of preferred 
parts and materials. 

2. Parts selection and authorization process. These are the procedures for 
authorizing new parts to the preferred parts selection list. The procedures will 
identify the IPT or organization responsible for authorizing parts for use and 
the structure and membership of the IPT. The procedures will also include 
criteria used to ensure suitability of parts intended for use in the required 
application; order of preference used in considering new parts, and procedures 
for notifying associated disciplines (inventory, purchasing, quality assurance) 
in case authorization of a new part should be included. 

3. Obsolescence management. This includes procedures for obsolescence 
management, which include proactive obsolescence forecasting for applicable 
part types (e.g., microcircuits) and plans for reacting and achieving solutions to 
obsolescence impacts as they occur and affect the program. 

4. Subcontractor management. This element addresses contractor procedures for 
establishing and maintaining subcontractor IPT participation to ensure 
satisfaction of the PM&P Program objectives. 


a: Part and supplier quality. This management element provides the requirements 





for assessing part suppliers and part quality, such as statistical process control 
data, audits, past performance, etc. 


6.  Part-level documentation requirements. Part-level documentation procedures 





need to be consistent with both the program’s logistic strategies and its need for 
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performance and re-procurement documentation at the intended level of logistic 
support. 

7. Substitute and alternate part procedures. This section addresses the process for 
the management and documentation of parts, other than those on an as-built or 
as-designed parts list. The program needs to be consistent with the intent and 
application of other disciplines (e.g., reliability, configuration management, 


quality, logistics, etc.). 


8. Customer-contractor teaming. This part of the management plan addresses 
how customer teaming is provided so that process insight and program 
verification (i.e., IPT participation, technical interchange meetings, as-built, 


and other parts lists) are allowed. 


I. COMMERCIAL PM&P PROGRAM PRACTICES 


The American Institute of Aeronautics and Astronautics (AIAA) has established 
AIAA R-100 [Ref. 25], a document that is a collection of recommended practices for 
managing parts and materials. The AIAA standard establishes a management approach 
that is consistent with the current business environment. The document strategy is to 


manage risk up-front by selecting the right part for the intent application. 


The standard emphasizes that a PM&P Program will deal pro-actively with parts 
and materials management issues and provide guidance for developing comprehensive 
strategies to manage cost and schedule risk via an IPT process. Figure 7 depicts the IPT 
process. The design process includes design margins, life cycle cost, technology 
insertion strategy, technical support, parts selection, and validation, all of which are 


addressed concurrently rather than sequentially. [Ref. 25] 
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Figure 7. PM&P Management IPT Overview [From Ref. 25] 


AIAA R-100 also establishes a design process. The flow diagram (Figure 7) 
illustrates that the interrelationships between the critical key elements of AIAA R-100 
must be addressed concurrently to achieve the “best practice” selection of parts, 
materials, and documentation required for the design. Figure 8 is important because it 
shows not only the key elements that must be considered, but also that the objective of 
this design process is to evaluate inputs from all the key elements concurrently and to 
then select the parts and materials that satisfy the product requirements. Figure 8 is 
helpful to the DoD because it demonstrates that there is a commercial “best practice” 


document with a parts and materials design approach. 
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Figure 8. Commercial Best Practice Part Design Process [From Ref. 25] 


SYSTEM- LEVEL RELIABILITY AND ITS IMPACT ON THE PM&P 
PROGRAM 


Section D mentioned some of the impacts of selecting a flawed system-level 


reliability requirement. The impacts of this requirement are far-reaching, even down to 


the individual part level. Once the reliability requirement has been established, it will be 


allocated among the various subsystems, units, assemblies, and so on. For a system with 


a high reliability requirement, this means that the parts that make up the system will need 


to have even higher reliability. 


For example, consider a system requiring 90 percent reliability. If the system 


were modeled with six components, each component would require a reliability of greater 


than 98 percent, if evenly allocated. At this high reliability level, the system becomes 


very sensitive to even one component not meeting its allocation 
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An example of the effect at the part level is the U.S. Army JAVELIN Missile. 
The guidance electronics circuit assemblies have a reliability allocation of about 99 
percent, which includes about 120 microcircuits and semiconductors. Therefore, even if 
everything else is perfect (no solder joint, printed wiring board, or passive part failures), 
each microcircuit’s and semiconductor’s part reliability must exceed 99.99 percent. [Ref. 
22] Now consider a system such as an airplane or helicopter, which has thousands 
instead of hundreds of such parts. Part reliability must be higher than system reliability, 
and even much higher depending on the number of parts; and failure to meet allocation 
on even one part would reduce the overall system reliability. The system-level reliability 
requirement is critical because once it gets allocated to the lower levels, it can easily 
drive the design and test at the lower levels of assembly (such as the individual piece 
parts) to unreasonable levels. 


K. CHAPTER SUMMARY 


In this chapter, the researcher described, first, the current reliability trend in the 
Army and, then, the PM&P Program. Background on how a reliability requirement is 
established on a weapon system included the origination of the requirement, how it is 
allocated, and the impact of establishing a flawed requirement. The relationship of 
reliability to logistics and life cycle costs was demonstrated. The benefit of “designing 
in” reliability was explained, along with the impact that system-level reliability can have 
on the individual parts and materials. The chapter covered former and current DoD 


policies and guidance, as well as commercial PM&P initiatives. 
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lil. DATA 


A. INTRODUCTION 


This chapter identifies and discusses a variety of issues, common practices, 
concerns, and real-world experiences of subject matter experts (SMEs) who work on 
PM&P Programs. Data are presented from SMEs who work on programs that include 
missiles, aircraft, and space systems in various stages of development. The data were 
gathered through telephone interviews, using questions based on the literature review and 
the background research on PM&P Programs described in Chapter II. The questions 
were designed to draw out the experiences and practices of each SME on managing 


PM&P Programs. 


This chapter is organized as follows: first, it discusses the general methodology 
and the process used to conduct the interviews; second, the chapter presents the interview 
question responses, grouped by common themes, and, where appropriate, provides them 
in tabular form; and, finally, it presents conclusions. Note that, with the exception of the 
definitions in Table 3, the source for the tables found in this chapter is the author, who 
based them on responses to the interviews. 


B. METHODOLOGY 
1. Overview 


This thesis focuses on optimizing the use of highly reliable parts, materials, and 
processes—a key element in maximizing weapon system reliability. The objective is to 
develop a set of heuristics to assist future Program and Logistics Managers. The method 
for accomplishing this objective is to analyze the methods and beliefs of those who are 
considered subject matter experts (SMEs) in the field. Analysis of the tactics and 
strategies that can be used to maximize the inherent reliabilities of system components 
will not only help future Program and Logistics Managers, but will also assist the DoD in 
achieving reliability goals that remain elusive. Since the primary research question 
addresses “what” tactics and strategies can be used, the best way to answer the question 
may be through the use of in-depth cases. However, since in-depth cases were not 


practical for this thesis due to resource constraints, telephone interviews with SMEs, the 
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individuals responsible for implementing the PM&P Program, were conducted instead. 
[Ref. 26] The purpose of the interviews was to gather data in order to identify common 
themes. 


2. Development of Interview Questions 


The interview questions evolved from the secondary research questions. The 
questions and their relationship to the research questions are depicted in the following 


matrix. (Table 1) 
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Secondary Research Questions 


Related Interview Questions 





1. What technology, tools, and techniques 
are available to ensure the proper selection 
of reliable parts and materials? 


a. What design tools can be employed to 
ensure that reliability is “built in” early in 
the design of a weapons system? 





b. What methodologies and techniques are 
used to determine parts, materials, and 
processes compliance? 





c. What methodologies and techniques 
have succeeded, and do you have any 
examples? 





d. What methodologies and techniques 
have not succeeded, and do you have any 
examples? 





2. How has acquisition reform changed the 
way a PM&P Program is managed? 


a. How was a PM&P Program managed 
prior to acquisition reform? 





b. How did acquisition reform change the 
way the PM&P Program is managed? 





c. How did PM&P Program risks increase 
or decrease with acquisition reform? 





3. To what extent does commercial 
industry differ in its approach to part and 
material selection, and can the Military 
leverage these best practices to improve 
performance in military systems? 


a. What methodologies and techniques 
does commercial industry use to manage 
the PM&P Program? 





b. What commercial industry techniques 
can DoD adopt? 





4. How can a Program and Logistics 
Manager mitigate part and material risks? 


a. What are the top three things a Program 
Manager can do to mitigate part, material, 
and process risk and maximize reliability in 
the design? 





b. How much and what kind of input does 
the PM get from the Logistics Manager on 
PM&P Program issues? 





c. Could the level of input be improved? 
Why or why not? 





d. If there is input from the Logistics 
Manager, is it useful? 











e. Are there any examples (good or bad) of 
this interaction? 





Table 1. Relationship Between Research and Interview Questions 


3. Choice of Study Subjects 


Interview subjects were chosen based on convenience sampling. Although this 


technique may not be representative of the population as a whole, the information can 
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still provide some significant insights and be a good source of data. [Ref. 27] Interviews 
were conducted with individuals from the Army (U.S. Army Aviation and Missile 
Command), Navy (Naval Air Systems Command), and Air Force (Space & Missile 
Systems Center) and with contractors with whom they interface. Due to their high cost 
and the safety-related nature of their application, these particular systems require a high 
degree of reliability. Each SME has a minimum of ten years’ experience working with 
PM&P Programs on military weapon systems. A total of 11 interviews were conducted. 
This base of subject matter experts should provide a fairly representative cross-section of 
experiences in utilizing highly reliable parts, materials, and processes in weapon systems. 


4. Interviews 


Interview questions were distributed to each SME via email, along with 
information regarding the objectives of the interview. This email was then followed up 
with a telephone interview. Since the interview subjects were located throughout the 
country, in-person interviews would have been neither cost-effective nor timely. [Ref. 26] 
In order to ensure that the data were valid, the interview subjects had full editorial 
control. If the answers were recorded incorrectly, the SME was given full opportunity to 
make corrections, or else the data were not used. 


Questions were written in a manner to avoid justifications—that is, they asked 


rd 


"how" instead of “why.” Leading questions and “yes or no” questions were avoided, as 
well. The goal was to get the subjects to express their ideas and thoughts about the 


issues. Table 2 summarizes the steps used in conducting the study. 














1. Identify and analyze the primary research question. 
Conduct a review of reliability and PM&P Program literature. 

3: Develop interview questions for subjects to obtain data to answer primary and 
secondary research questions. 

4. Conduct a review of personnel who have extensive experience with PM&P 


Programs and issues. 





Arrange for and conduct telephone interviews with subjects. 





Analyze subjects’ responses for common issues. 





Compare subjects’ responses. 





Develop generalizations and summarize data. 











SO] SOTA 


Develop a set of critical strategies and tools to aid future PMs and LMs. 





Table 2. Summary of Methodology 


30 





C. TECHNOLOGY, TOOLS, AND TECHNIQUES 


The first group of questions focused on the technology tools and techniques that 
are used to select and control the parts, materials, and processes; how compliance is 
verified; and what methods were successful and unsuccessful. The first question asked 
about the design tools that can be employed to ensure that reliability is “built in” early in 
the design of a weapons system. The SMEs’ answers and their definitions are shown in 


the Table 3 below. 





Reliability Prediction Analysis - is simply the analysis of parts and components in 
an effort to predict the rate at which an item will fail. A reliability prediction is one 
of the most common forms of reliability analyses. 





Circuit Tolerance Analysis — is a technique, which, by accounting for component 
variability, determines circuit performance under a worst-case scenario, i.e., under 
extreme environmental or operating conditions. This analysis is particularly cost- 
effective in safety-critical applications and for products expected to operate in severe 
environments. 





Sneak Circuit Analysis — detects design flaws that escape reliability testing and 
impacts system performance by activating unwanted functions or impeding desired 
functions. It is crucial that sneak circuits are found as early as possible. Failure to 
do so can lead to costly late redesign or safety-related failures. 





Physics of Failure (PoF) Tools — is an approach that uses knowledge of root-cause 
failure processes to prevent product failures through robust design and manufacturing 
practices. The PoF approach proactively incorporates reliability into the design 
process by establishing a scientific basis for evaluating new materials, structures, and 
electronics technologies. The PoF approach encourages innovative, cost-effective 
design through the use of realistic reliability assessment. 





Part Derating — is the practice of limiting electrical, thermal and mechanical stresses 
on parts to levels below their specified ratings. If a part or material is expected to 
perform reliably, it must incorporate a conservative design approach using derating. 





Government Industry Data Exchange Program (GIDEP) - is a cooperative 
activity between Government and industry that reduces or eliminates expenditures of 
resources by sharing technical information essential during research, design, 
development, production and operational phases of the life cycle of systems, 
facilities and equipment. Proper utilization of GIDEP data can improve the 
reliability of systems and components during the acquisition and logistics phases of 
the life cycle. 





Prohibited Parts List — is a listing of parts deemed unacceptable by the PMB for 
use in a company’s products because of cost, quality, safety, etc. 
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Failure Mode, Effects and Criticality Analysis (FMECA)—or simply FMEA— 
identifies potential design weaknesses through systematic analysis of the probable 
ways (failure modes) that a component or equipment could fail. It is a disciplined 
technique that focuses the development of products and processes on prioritized 
actions to reduce the risk of product field failures, and it documents those actions and 
the review process. 





Fault Tree Analysis — is a deductive, top-down method of analyzing system design 
and performance. It involves specifying a top event to analyze (such as a fire), 
followed by identifying all of the associated elements in the system that could cause 
that top event to occur. 





Part and Material Selection List - is a part and material listing preferred for use in 
equipment design, which often contains descriptions, attributes, or application 
information. 





Product Change Control — consists of process and procedures that manage how 
changes are incorporated into a product. Product Change Control is a part of overall 
configuration management and uses review and release processes to enforce 
compliance with enterprise change policies. 





Finite Element Analysis - is the method of simulating a physical part or assembly’s 
behavior by building a model of the geometry, dividing it up into connected 
elements, applying loads and constraints, and then calculating deflections and 
stresses. 





Failure Reporting and Corrective Action System (FRACAS) - is defined as a 
closed-loop process for identifying and tracking root failure causes and subsequently 
determining, implementing and verifying an effective corrective action to eliminate 
their reoccurrence. The FRACAS accumulates failure, analysis and corrective action 
information to assess progress in eliminating hardware, software and process-related 
failure modes and mechanisms. 





Circuit simulation tools - simulates and verifies a design for logic and testability 
functions, timing behavior, and accuracy to other representations of the design 
(behavioral-level, discrete breadboard-level, etc.), and performance in the target 
system. 





Design Verification Test (DVT) - involves detailed tests to verify conformity to all 
applicable interface standards, as well as all vendor-specific specifications and 
diagnostic commands. Performed to validate designs, DVT is extremely rigorous 
and often subjects the product to heat/cool, electrostatic, and EMI/RFI testing. 


Table 3. Design Tools [From Ref. 28] 











The purpose of the second question was to discover the methods and techniques 
used to determine parts, materials, and processes compliance. Table 4 provides a list of 


the responses. 
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Receiving Inspection 





Qualification Testing 





Environmental Stress Screening 





Reliability Growth Testing 





Process Controls 





Reliability Predictions 





Prohibited Parts List 





Evaluation of Contractor PM&P Program 





Data Review 





Surveys and Audits 





Destruct Physical Analysis (DPA) 








Communication 








Table 4. Techniques used to determine compliance. 


The third question dealt with the methods and techniques that succeeded. This 


section groups the responses 


Communications, and (c) Data analysis. 


into the following categories: 


(a) Testing, (b) 


These categories were based upon how the 


interviewees addressed the topic. Excerpts from the interviews are provided below. 


a. Testing 
The following interview excerpts deal with the topic of testing. 


When in doubt, test the parts. Example: on Program XXXX, the supplier 
qualification data for a SCR-Diode module showed the piece part 
qualification had been done on a sample size of only ten pieces, and 
Autoclave had not been done at all. We did a full qualification test per the 
requirements, with sample size of 50 for all tests except life test, which 
was done on 80 parts. Ten percent of the modules failed temperature 
cycling and autoclave due to what appears at this time to be an inherent 
design problem. The sample size of ten pieces in the supplier qualification 
in only one of the tests failing was not adequate to identify this problem. 


Plastic encapsulated microcircuit qualification testing and unit 
qualification testing [are important] because [they] screen out weak 
suppliers. 


[We performed] plastic encapsulated microcircuit sequential testing; this 
was example of [an] independent test that allowed programs to separate 
the inconsistent vendors from the consistent ones. 


b. Communications 


The following excerpts discuss communication. 


Require a PM&P Program that clearly outlines the procedures necessary to 
have in place to ensure the use of quality components that will meet the 
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overall system performance requirements. There needs to be an expert in 
this field as part of the team that develops the plan and should be involved 
with any modifications to the plan. There is a company that has recently 
developed a plan that covers all three services and was put together with 
input from representatives of all three services. It is up to each program to 
ensure the plan is being implemented properly. 


We only get data when we fail. It’s difficult to tell what works; we have 
tracking of what does not work. Most times, the problem is the contractor 
changing things and not telling the prime. Adherence to processes and 
communications is the key. 


Supplier Surveys and Audits are especially successful. If nothing else, 
they give the Program Office and contractor personnel a chance to meet 
face-to-face to discuss issues and problems. 


We are also having success with conducting supplier-management surveys 
and audits. It’s been very successful at getting insight into what the 
contractor does, especially for keeping track of process changes. 

Cc. Data Analysis 

The following excerpts have to do with data analysis. 
IPT working groups are needed to analyze parts and materials, methodically 


reviewing parts against the application requirements. 


You basically have to know if the part or material is right for the 
application and if it fits the mission profile. 


Proactive Integrated Product Teams (IPTs) are one of the best vehicles to 
ensure product performance, reliability and repeatability. This method 
ensures all parties are addressing these issues and are staying connected. 
However, this only works when all functionals participate and coordinate 
their efforts. 


To ensure reliability, you will have to get down to the supplier and get into 
his processes to ensure piece part reliability. 


The fourth question dealt with the methodologies and techniques that have not 
succeeded. The responses to this question were interesting because most of the answers 
had nothing to do with tools and techniques. Instead, most of the responses were about 


communication and the lack thereof. This section groups the responses into the following 
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categories: (1) Communications and (2) Design Process. Excerpts from the interviews 


are listed below. 


d. Communications 
The following excerpts regard communication. 


That’s tough because a lot here can depend on how you approach you 
counterpart. An aggressive approach usually yields a level of distrust and 
standoffness. 


Contractor Management of their own PM&P without Government 
oversight [is needed]. 


We get into trouble when we buy a “black box” and don’t pay attention to 
what PM&P is in it. An example is the use of pure tin. For most part 
types it’s not allowed due to tin whisker growth. On one program, the 
subcontractor had tin in his black box, and he did not inform the prime 
contractor. The prime also did not verify the materials in the black box. 
The box failed and caused the failure of a high-profile, high-cost missile 
test. 


[There is a] lack of concurrent engineering (the IPPD process is not 
always rigorously applied). 


[We] rely on contractors to provide a system that meets a performance 
requirement without insight. 


I know of no problems with the tools or techniques. There was a problem 
with the subcontractors. They were making changes to processes without 
telling the prime contractor. 


e. Design Process 


The following responses address the design process. 


Destructive Physical Analysis (DPA), when not accompanied by a 
comprehensive PoF model, may not be adequate . . . to identify potential 
problems in parts. 


The most important thing is paying attention to detail. 


The most prominent practice that comes to mind is when the contractor 
uses the “design as you go” methodology. The design NEVER gets 
locked down, and engineers are standing behind the operators changing 
layout and components as the CCAs are fabricated. This often results in 
shorts, opens, and numerous jumper wires and wrong parts. 
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Not using the design techniques I mentioned before. 


Contamination of plastic encapsulated microcircuits that will cause a latent 
failure, intermittent problem, or system failure. The soldering process (part to 
board, usually from a poor-cleaning process) usually causes this. 


D. ACQUISITION REFORM 


The second series of interview questions focused on the effects of acquisition 
reform on PM&P Programs and how PM&P Program risks increased or decreased with 
acquisition reform. The first question inquired how a PM&P Program was managed prior 
to acquisition reform. There was almost complete agreement in the responses to this 
question. All of the subjects responded that strict control was established through 
military specifications and standards that were contractually required. Below are the 
responses. 


MIL-STD-965 was used. Parts needed to be selected off a list, and if it 
was not on the list, a “Non-standard Part Request” was submitted to the 
Government. Also, there was a full level-HI technical data package, and 
all the reliability tasks of MIL-STD-785 were required. 


Use of MIL-STD-965. Use of military specification parts/established 
reliability parts was the first order of selection. Standardization was 
emphasized. 


MIL-STD-965. Part Engineers would review the final parts selected, for 
compliance to standard requirements at the end of the line. 


Control, Control, Control. 


Contractors were required to adhere to military specifications and 
standards as required in their contracts. 


Design guidance called MIL-STD-11991 and MIL-STD-454 were used. It 
specified what PM&P to use and what not to use. MIL-STD-965 was used 
for standardization; it also required the use of military specification parts. 
It did not allow commercial parts. There were a lot of data requirements. 
Data would have to be sent in to the Government for review and approval 
as official data submittals. 


Control of a level-III TDP allowed for more Government control/insight 
into the parts used. 
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MIL-STD-965 was required. 


[The] MIL-STD-965 requirement was invoked in the contract. This 
required the contractor to submit all parts, and some materials, to be used 
in design, (except those parts contained in the Government-furnished 
baselines or those previously approved by contract), for approval. The 
problem for the contractor was the time it took to receive a part approval, 
which included subcontractor part submission to the prime contractor and 
then prime contractor submission to the Government (not done 
electronically). 


We used two documents, MIL-STD-1546 and MIL-STD-1547. MIL-1546 
was like MIL-STD-965, but more stringent. MIL-STD-1547 was more 
technical; it specified the exact screens, tests, PDA, lot size, etc. These 
two documents told contractors how to manage the PM&P Program. 
Contractors were required to have a PM&P Program, and they had to have 
a PM&P Control Board. 


All the typical quality and reliability Military Specifications and Standards 
were levied in the contract. The contractor was required to adhere to the 
military specifications and standards as required in the contract. This 
included requirements such as MIL-E-11991, MIL-STD-965, MIL-STD- 
785, MIL-STD-1629, MIL-HDBK-189, and MIL-HDBK-781. 


The second question tried to ascertain how acquisition reform changed the way 
the PM&P Process is managed. All of the responses reflected the fact that military 


specifications and standards could no longer be required. The responses are shown below. 


It shifted from an imposition of standards to a process whereby the 
contractors describe their processes and procedures. If these processes 
and procedures do not meet the goals of the PM&P Program, the 
differences are negotiated to resolution. 


[We] cannot use MIL-STD-11991 anymore. Contractors now have to 
each create their own PM&P design guidance. Contractors still have to 
get data to make decisions and some still submit data, but it is on a 
voluntary basis now. If a teaming arrangement is in place, most data is 
(sic) obtained informally, but sometimes travel is required to go look at it 
because they won't send it out. 


Parts and materials engineers must become more knowledgeable and must 
be multi-disciplined. One of the most important aspects of parts 


engineering now is to be a part of the early component-selection process. 
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Parts Engineering has become an up-front activity, where in the past, 
under MIL-STD-965, it had become an end-of-the-line activity. 


With acquisition reform, the Government can somewhat control the 
process in which parts and materials are chosen and tested, but there is not 
the control down to the piece part level that was there before. 


Less documentation required, screening and qualification levels lowered 
or non-existent, allowance of plastic encapsulated microcircuits in military 
systems, significant increase in parts obsolescence, and dependence on 
product change notices. 


Too liberal. The response was to throw away everything we knew and 
proceed blindly. The move to acquisition reform was not well thought 
out, and companies reacted blindly to strip parts support from design 
activities 


It changed from a MIL-STD-965 requirement to a performance-based 
statement of work type of requirement (examples contained in MIL- 
HDBK-512). The contractor would respond to the SOW requirement with 
his internal part management process. NAVAIR would review the process 
to insure that all of the standardization, reliability, and supportability 
issues were addressed. To ensure compliance, NAVAIR would have to 
maintain insight through access to the contractor’s part database. 


From a contractual standpoint, there were no more “how to” requirements; 
neither were there any more technical process requirements. The 
underlying philosophy is that the contractors have their own PM&P 
Program. There is some up-front assessment prior to contract award, but 
there is no longer a standard process for all programs. Also, the contractor 
has much more flexibility to express part-level test requirements. 


Military specifications for PM&P Program management were no longer 
imposed. All the plans and practices that were established in the military 
specifications were gone, and now each contractor had to manage and/or 
develop their own PM&P requirements. Use of commercial parts was 
emphasized. 


The contractor is now given a performance specification in which he is 
required to provide hardware that complies with the performance 
requirements as stated in the contract. The contractor is allowed to do 
whatever is necessary to ensure the hardware meets the requirements 
stated in the contract. 
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By tailoring contractual language to detail performance requirements in 
such a way that the contractor cannot claim that we are, in fact, telling him 
“how to” build our product. 


The third question tried to determine if risks relative to the PM&P Program 
increased or decreased with acquisition reform. There was almost complete agreement 
that the risks had increased. Below are the responses. The lone exception was an 
individual who worked on a weapon system that was still using military specification and 
standards requirements. The program had negotiated an agreement with the contractor to 
keep using the requirements, and that subject stated that there was no change in risk. The 


responses are listed below. 


Risks are especially high in organizations where Parts Engineers are not 
knowledgeable and are not involved in the component selection process. 


Risk has increased, but it is dependent on the Project and the people 
working in the project. Communication is the key .. .. 


Risk increased due to [the] answer above, but risk also decreased due to 
use of IPPD process (better coordination between disciplines, more 
tailoring of selected capabilities to application requirements). 


The risks increased. We relinquished control (as required by no contract 
requirements), and the risks increased. 


Risk has increased because no standard methodology of parts management 
is in place. 


In my opinion, risk has increased significantly. The Government tried to 
balance cost and resources and it did not work. The contractor needed to 
modify processes to lower the bid price, so they streamlined the 
specifications and standards, along with personnel and processes. 


Risks to the Government increased. This was due to the fact that we only 
knew what the contractor did, as far as part decisions, after the fact, and in 


some cases, due to a lack of personnel, not at all. 


Risk has not increased or decreased at all on the program [where] I work. 
We still use all the military reliability requirements. 


Risk has increased, but it is dependent on the Program Office and the 
people working there. Communication is the key; if it is good, the risk is 
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lower, [and] if it is bad, then risk has increased due to the lack of oversight 
and inadequate peer review. 


Risks increase for contractors who are not willing to negotiate towards the 
goals of the PM&P Program; the risks decrease for contractors who are 
willing to negotiate in good faith in order to deliver quality hardware. 


Definitely much more risk. It is next to impossible to get a contractor to 
do anything outside of their normal operating procedures (which are 
usually under the auspices of “best commercial practices”) without hard 
contractual requirements to back them up. 


E. COMMERCIAL INDUSTRY 


The third series of interview questions focused on the methodologies and 
techniques that commercial industry uses to manage parts, materials, and processes and 
what commercial industry techniques, if any, the DoD could adopt. The first question 
asked about the methodologies and techniques commercial industry uses to manage parts, 


materials, and processes. These are the responses: 


They do not really do anything differently. They have more of the ability 
to start research or perform special testing (if necessary) on new materials. 
A lot of times, commercial industry can be more tolerant of failures 
(depending on market impact), such as poor production yield. 


I cannot comment, as I have no experience with commercial OEMs. 


Not sure. 


They depend on Process Change Notices (PCNs) from part suppliers. 
They also do more first-article qualification at the part level. 


They are selective in suppliers they deal with and they use “agreements” 
to control proliferation. They also manage their suppliers much more 
efficiently than military. 


If commercial industry has processes, they would closely resemble those 
of the Defense Contractors. Defense Contractors only developed their 
processes because of MIL-STD-965. 


Commercial companies certainly do PM&P Management, but it is a totally 
different business. The make use of strong incentives and high volume 
production. The incentive is the opportunity for follow-on business. The 
big difference between commercial and military is that commercial 
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industry uses high volume and [a] low number of part types. The military 
is just the opposite. This gives the commercial industry a lot more 
leverage. 


Some companies are trying to develop their own design guidance. The 
problem is that it is high-order and not detailed. However, some good 
documents do exist for guidance, including the AIAA R-100, SAE R&M 
100, AEC QS9000 documents, and the Stack International documents 
(these document are based on the old military specification and standards). 


In a general sense, they each have company standard processes that they 
use. But each company and each division within a large company all use 
their own peculiar process. Some don’t even manage PM&P. Some still 
use military specifications, but most have abandoned due to cost. 


When it is to their financial advantage, the contractors will adhere to select 
guidelines contained in the PMP. It has been my experience, however, 
that they will always do what is the fastest, cheapest and easiest process. 


I will take a pass on this one; I don’t have insight into commercial 
processes. 


The purpose of the second question was to see if the respondents thought that 
DoD could adopt any commercial industry techniques. As might be expected after 
reading the responses to the above question, the subjects did not think that many 
commercial techniques could be utilized. Five either did not know of any commercial 
techniques or could not think of any that DoD could use. Below are the responses of 


those who did have an answer. 


Concurrent engineering and IPPD, which we are using. 


Contractors have the ability to quickly obtain expertise from experts when 
they have PM&P issues. There are a number of consulting firms that will 
provide this service. However, we have teamed with most of our 
contractors to create basic methodologies based upon a combination of our 
lessons learned and their lessons learned information. I don’t know; I 
don’t see a lot of things different. We do try to utilize any industry 
document that we feel is applicable to missile systems—e.g., AEC, EIA, 
Stack international, etc. Also within other Government agencies—for 
example, the Department of Energy has just forwarded us a preliminary 
design guide that one of their subcontractors has developed for the nuclear 
people. 


4] 


In the past, DoD has not been willing to pay the upfront cost needed to set 
a “commercial” infrastructure. Too many rules and regulations. It’s not 
politically correct. Until this “control” without payment idea is changed, 
it will be difficult to adopt the buyer/seller relationship, guaranteed 
procurements, standardized products and block upgrades. 


None that I'm aware of. 


It’s difficult to say. The consumer product is very different from what the 
military requires. The military requires very robust, long lasting parts. 
The commercial world does not necessarily test parts, even though they 
guarantee parameters. The military requires actual testing and not just 
guarantees. It’s also hard to leverage off the commercial world because 
they design to a different level. Some projects are doing “up-screening” to 
get parts that perform adequately. 


This all depends on the DoD system. If the DoD is developing a system 
that is typical to the environments and lifespan of commercial hardware, 
then we can adopt many of the industry techniques. If, however, we are 
developing a system that is to be deployed in [a] very harsh military 
environment with fairly lengthy life cycles, then we need to be very 
careful what techniques we allow. We are currently allowing the use of 
certain test data from microcircuit manufacturers to provide confidence 
that the manufacturing process used is sufficient to produce quality 
components. In some cases, manufacturers do not provide sufficient data 
to warrant this, and further tests are needed to provide that confidence. 


F. MITIGATING PM&P RISK 


The fourth interview question was about things a Program Manager can do to 
mitigate part and material risks. The question asked the top three things a Program 
Manager can do to mitigate part, material, and process risk and maximize reliability in 
the design. The responses below list the interviewees' suggestions. 


1) Recognize that they need to perform qualification testing at the part and 
material level; 2) require the contractor to implement a comprehensive 
FRACAS down to the part level; 3) monitor suppliers for changes to 
PM&P that would affect performance. 


1) Plan for adequate funding in the reliability area and don’t take it. 
Programs usually take money from test and reliability for other priorities; 
2) Have project office personnel participate in all (or as many as possible) 
IPT meetings; 3) emphasize communication at all levels. Some programs 
keep too much information “close hold” and do not have adequate peer 
review with subject matter specialists. 
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1) Ensure that highly knowledgeable commercial component experts are 
involved in the early concept process and the early part selection process; 
2) allocate funding for independent testing of piece parts, and let the 
Component Engineers know that there are no barriers to testing of parts 
where they feel it is required; 3) require that the entire part review for each 
part be documented; perform QC audits of the PM&P review process early 
in the Design, and periodically throughout the design process. 


1) Develop a comprehensive PMP plan as early as possible in the system 
life cycle; 2) maintain support from PMP “experts” throughout the 
program; 3) make sure the performance spec has adequate requirements to 
enforce the PMP program. 


1) Provide adequate budget for detailed part assessments; 2) enforce part 
test and qualification requirements; 3) use a piece part level FRACAS that 
requires root cause analysis down [to] the part level. 


1) Establish the PMP rules and authority immediately after contract award; 
2) empower the PMP organization to do its job within the boundaries 
outline by the first bullet; 3) support the PMP activities within the 
engineering design organization. 


1) Write good contract requirements; 2) invoke a Parts Management 
Program (PMP) requirement in the contract; 3) assign a PM&P Program 
IPT lead to insure that all part technical issues, including "lessons 
learned," are addressed. 


1) Know and understand your part and material application; 2) know and 
understand the inherent design of the parts and materials; 3) know your 
part and material suppliers. 


1) [Have a | Parts Management Plan; 2) use and support effective Testing 
Techniques (part level, board level, DVT); 3) use adequate design tools. 


1) He needs a PM&P Plan that includes testing at the part level; 2) need to 
perform construction analysis on parts to verify changes; 3) need a closed 
loop FRACAS to avoid long-term reliability problems. 


1) Allow early part expert review of selection of parts and materials; 2) 


fund qualification testing of parts and materials; 3) fund root cause and 
corrective action of part failures early in design and during production. 
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G. INTERFACE WITH LOGISTICS 


The last series of interview questions focused on the type and quality of input a 
PM gets from the LM on part, material, and process issues. The first question asked how 
much and what type of input the PM gets from the LM on part, material, and process 
issues. There was overwhelming consensus that there is little support from the LM. 


Below are the responses. 
Logistics does not do much up front. 
I work several programs, and I only know of one program that gets input 
from the LM; otherwise it is non-existent. The type of input usually 
revolves around supportability issues. Actually, what I am trying to say is 
I only have one program where the logistics people work closely with 
other engineers, such as this office, to provide the best advice to the PM. 
Usually logistics works largely independent of everyone else, it appears. 
All programs get input from logistics, but on other programs they don’t 
always seem to work with other organizations. 
I cannot comment on this, as I have not interfaced with Logistics. 


Very little that I have seen. 


They provide very little input, only minimal input of field failure data and 
standardization. 


None that I am aware of. 


These questions are difficult to answer because generally there is not any 
input from the Logistics Manager. 


Don’t know, our organization does not get much input from logistics. 
Logistics is not really involved. 
The space world does not do sustainment; once it's launched, that’s it. 


Usually nothing. 


The second question asked how the level of input could be improved, and why. 
Again there was an overwhelming consensus; this time, most respondents thought the 


level of support from the LM needs improvement. Below are the responses. 
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Yes, they could provide the logistics impacts of the early design. 


Yes, it definitely could because when the LM understands the importance 
of reliability and when he provides good support and works with other 
functional support people, the PM gets the better recommendations. 


Yes, it would help with supportability type issues. 


Yes, he needs to be involved. He is the one that has all the lessons learned 
from the field. Education of the logistics people is a big issue. 


Yes, it could be improved, but it might be hard to get logistics involved. 
In the past, they have offered support on issues such as transportability. 


I cannot comment on this, as I have not interfaced with Logistics. 
Needs to be improved. 


Yes, they could provide better feedback at the part and material level from 
the field and depots. 


Yes, he needs to be a big part of this because obsolescence is such a big 
issue with short-life-cycle items like computer parts. 


I am not sure. Perhaps they could provide input into specification 
requirements. 


It is a matter of philosophy of the program. I believe with the level of 
integration and pin count density has lead the logisticians to stop at the 
LRU/CCA level (since a depot probably should not be performing piece 
part removal and replacement). 


The third and fourth questions asked the respondents to say if the input from the 
LM is useful and to give examples (good or bad) of that input. The subjects typically did 
not answer these two questions because of the lack of contribution from the LM. In the 
rare times that the LM offered input, some thought it useful and some did not. Below are 


the responses regarding input from the LM. 


Not really. The logistics manager appears to be lacking the technical skills 
needed to impact the early design. 


Yes, very useful. Like I said before, I think it is most useful when the 
logistics works closely with reliability, engineering and other groups and 
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the projects. The LM offered input on how the equipment would be used 
by the soldier in the field. After discussions with product assurance 
people and other engineering, it caused a design change that would not 
have otherwise been made. 


All feedback from Logistics is useful since it allows the PM to alter the 
design, if necessary, to account for issues seen in the field. 


Input is only minimally useful at the LRU or subsystem level. 


At my level of insight, typically no. When it comes to obsolescence, the 
length of time it takes the logistics community to determine what assets 
they are in need of, the window of opportunity has closed. 


H. CONCLUSION 


Eleven SMEs in the parts, materials, and processes functional area described their 
experiences working on weapon systems. The SMEs represented programs from the 
Army, Navy, and the Air Force. Each person discussed characteristics of the program or 
programs he works on. The responses were indicative of their personal experience in 
working on PM&P Programs. The next chapter contains an analysis of these data and an 


attempt to develop a set of heuristics to assist future Program and Logistics Managers. 
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IV. ANALYSIS 


A. INTRODUCTION 


This chapter analyzes the data presented in the previous chapter. The analysis is 
based on the interview responses of participating subjects and is structured around the 
same topics as in Chapter III. The analysis consists primarily of comparing the survey 
responses to one another, searching for common issues and identifying trends. The 
researcher uses this analysis to identify strategies and tactics for optimizing the use of 
highly reliable parts, materials, and processes in order to maximize system-inherent 
reliability in the life cycle. 

B. CLASSIFYING INTERVIEW DATA 


Analysis of the interview responses was conducted according to the classification 
scheme presented in Chapter III. The interview responses were organized in relation to 
the secondary research questions, and a comparison was made to yield trends and 
commonality. In general, the secondary research questions pertain to five categories: 1) 
the technology, tools, and techniques available to ensure the proper selection of reliable 
parts and materials; 2) the way in which acquisition reform has changed how a PM&P 
Program is managed; 3) the extent to which commercial industry differs in its approach to 
part and material selection; 4) how a Program Manager can mitigate part and material 
risks; and 5) how a Logistics Manager interfaces with the PM&P Program. 

C. ANALYSIS OF KEY ISSUES 


1. Technology, Tools, And Techniques 


What enablers allow the selection and verification of highly reliable parts, 
materials, and processes? From the data gathered, there appear to be numerous tools and 
techniques that can be used to ensure that the design of a weapon system is maximized 


for reliability. 


Table 3 of Chapter III listed and defined the technologies, tools, and techniques 


that were mentioned in the interviews. Table 5 below lists the six most frequently 
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mentioned responses (mentioned by at least four of the subjects). Definitions of these 


tools are listed in Table 3, Chapter III. 





Top Six Design Tools That Are Used To Ensure That 
Reliability Is “Built In” Early In The Design Of A 
Weapons System 





Reliability Prediction Analysis 





Circuit Tolerance Analysis 





Sneak Circuit Analysis 





Physics of Failure Tools 





Part Derating 





Government Industry Data Exchange Program 


Table 5. Top Six Design Tools 











Proven and/or disciplined design approaches are a critical factor in the reliability 
success of a product. The tools and techniques in Table 5 were identified in the 
interviews as most utilized by programs in weapon system design (part and material 


selection and control). 


The four most frequently mentioned methodologies and techniques used to 
determine compliance are listed in Table 6. Interestingly, all but two of the interview 
subjects stated that some sort of testing was necessary to verify compliance. One even 
responded with, “Testing, testing, testing, (i.e., all types: board-level, part-level, 
independent testing, DVT, and special testing), all types and levels are important.” The 
responses were a strong indicator that testing at all levels is critical to verify part and 
material compliance. All of the interviewees mentioned that either testing or some type 
of data review is necessary ensure compliance. Even the establishment of a PM&P 


Program was secondary to testing. 
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Top Four Methodologies And Techniques Used To 
Determine Part, Material, And Processes Compliance 





Testing (i.e., Environmental Stress Screening, 
Qualification Testing, Design Verification Testing, 
Independent Testing) 





Evaluation of Contractor PM&P Program 





Data Analysis and Review 





Surveys and Audits 











Table 6. Top Four Verification Methods 


The answers to the question about successful methodologies and techniques 
provided the common themes listed in Table 7. Adequate communication was the answer 
mentioned by most people. Everyone responding also said something about performing 
some type of testing or data review. There already is a recurring theme that testing needs 


to be part of a PM&P Program. 





Successful Methodologies And Techniques 





Communication through IPTs 





Part and Material Testing. 





Surveys and Audits 











Table 7. Successful Methodologies and Techniques 


The subject of methodologies and techniques that have not succeeded did not 
yield the type of answers expected. This was intended to be an assessment of the tools 
and techniques that were identified in Table 3 of Chapter III. Only one of the answers 
had something to do with tools and techniques; most of the responses were about 
communication and the lack thereof. Though unexpected, the answers were, 
nevertheless, consistent with the answers to the previous question. One interview subject 


stated that “the lack of concurrent engineering” stood out in his mind as the biggest issue. 
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Another response was that “communications is vital.” Someone else said, “It depends on 
how you approach you counterpart. An aggressive approach yields a level of distrust.” It 
was evident that all the respondents believe communication to be one of the more 
important aspects of a successful PM&P Program. In a separate discussion, one 
individual remarked: 


I do not believe there are any tools you can use to build in reliability. 
Select the right part from the right supplier based upon all the available 
data you have . . . to ensure reliability, you will have to get down to the 
supplier and get into his processes to ensure piece part reliability. 


This and the other statements strongly imply that communication between all the 
parties involved is the most critical factor. 


2. Acquisition Reform 


This section analyzes the effects of acquisition reform on PM&P Programs. 
There was complete consensus about the way PM&P Programs were managed prior to 
acquisition reform. Consistent with the literature review, the common theme was that 
strict Government control was established through military specifications and standards. 
MIL-STD-965 was the prevailing document that compelled the use of a standard process. 
Along with the requirements of MIL-STD-965, one person said that “there were a lot of 
data requirements, and data would have to be sent to the Government for review and 


approval.” 


For changes after acquisition reform, there was universal agreement that military 
specifications and standards could no longer be used. This was also as expected from the 
literature review. Because of the change to performance-based contracting, the level of 
control enabled by military specifications and standards was no longer there. The focus 
shifted from controlling the design of the product to having insight into the contractor’s 
processes. The answers also indicated an increased dependence on the ability to 
communicate. No longer could a program sit back and wait for data to be delivered. Ifa 
program wanted insight at the part and material level, personnel had to be proactive and 
get the information by participating in the IPT process. The result of not being able to 
use military specifications and standards can be grouped into three categories: 1) there is 


no longer one uniform process with strict Government control; 2) there is an inability to 


50 


utilize the lessons-learned contained in the military specifications and standards; and 3) 


requirements had to be specified in performance-based terminology. 


On the question of post-acquisition reform risk, there was an overwhelming 
consensus that acquisition reform had increased risk. All of the survey respondents were 
in complete agreement, except for one individual, who said that risk had not changed. 
His program had negotiated with the contractor to maintain the requirements that were in 
the cancelled military specification. Even without taking this into account, its obvious 
that acquisition reform is not held in high regard in the parts, materials, and processes 
arena. The most interesting part was that even the contractors who were interviewed also 
agreed that acquisition reform had increased risk. This is interesting because one aspect 
of acquisition reform was to provide contractors relief from Government-unique terms 
and conditions. [Ref. 29] One of the contractors stated, “Too liberal. Response was to 
throw away everything we knew and proceed blindly. The move to acquisition reform 
was not well thought out, and companies reacted blindly.” Another contractor said, 
“Design Engineers have other priorities which often conflict with the selection of highly 


reliable parts.” 


Through all the interviews, one positive theme emerged from the discussions of 
acquisition reform. That was the use of IPTs. Concurrent engineering and the use of 
IPTs were mentioned as a method that has reduced some of the risks that were created. It 
did this in two ways: 1) by increasing the flow of communication, and 2) by involving 
parts and materials engineers in the earlier stages of the design. 


3. Commercial Industry 


This section analyzes the techniques that commercial industry uses to manage 
parts, materials, and processes and which, if any, DoD could adopt. The responses here 
varied greatly. A number of the interview subjects stated that they were unfamiliar with 
commercial industry's practices. It is apparent that commercial industry has not, as 
envisioned, stepped up and developed “best practices” or initiatives to replace part and 
material military standards and specifications. The common theme among the 
individuals who did answer this question was that commercial industry does not really do 


anything differently from the military. In fact, it appears that commercial industry 


51 


leverages off the DoD for part and material requirements and testing. When something is 
done differently, such as special testing, it is because commercial industry operates in a 
more flexible environment. It is also apparent that commercial industry is a different 
type of business. Because commercial businesses are a bigger part of the market, they 
have more leverage with suppliers. Also, depending on the market impact, they also can 
be more tolerant of failures. Due to safety and readiness reasons, the military is typically 


not as tolerant of failures. 


As might be expected from the above responses, the subjects did not think that the 
military could use very many commercial techniques. In fact, the first common theme 
that emerged was that the military already leverages commercial practices, such as 
concurrent engineering and IPTs. The other theme is that the DoD does not want to use 
some commercial practices because of the differences between the two markets. The 
military application environment is much more severe than the typical commercial 
application. One person said, “If we are developing a system that is to be deployed in [a] 
very harsh military environment with fairly lengthy life cycles, then we need to be very 
careful what techniques we allow.” Overall, there seemed to be agreement that the DoD 
was already leveraging off commercial practices where it was practical. 


4. Mitigating PM&P Risk 


This section analyzes the tools and practices that a PM can use to mitigate part, 
material, and process risks. The purpose of this question was to find synergy among the 
experts with regard to managing a highly reliable PM&P Program. Table 8 provides a 


summary of the most frequent responses. 
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The Top Things A Program Manager Can Do To Mitigate PM&P 
Risk And Maximize Reliability In The Design 





Provide adequate funding for a PM&P Program that includes testing at 
the part and material level. 





Develop a comprehensive PM&P Program as early as possible in the 
system life cycle. 





Ensure that adequate parts, materials, and process requirements are 
placed in contracts. 





Involve PM&P experts early in the process. 





Implement a FRACAS down to part and material level. 





Monitor and emphasize communication with suppliers. 
Utilize IPTs, including a PM&P IPT. 
Table 8. The Top Things A Program Manager Can Do To Mitigate PM&P Risk 














It was interesting that none of the responses had anything to do with the tools and 
techniques mentioned earlier. Three main themes developed: 1) establish a PM&P 
Program early in the system life cycle that includes testing at the part and material level; 
2) institute strong management support (funding and enforcement); and 3) make 
communication between all parties a part of the process. 


5. Interface with Logistics 


This section analyzes how much and what kind of input the PM gets from the 
Logistics Manager in a PM&P Program. The purpose of this series of questions was 
twofold; it was an attempt to get an indication of the level of support being provided and 
then to find out how this support could be improved. From the list of responses in 
Chapter III, it is readily apparent there is little support from the LM on part, material, and 
process issues. Only one of the interview subjects even knew of a program where the 


LM was involved 


There was also a resounding consensus that the level of support needs to be 
improved. Quite a few interviewees commented that the LM has expertise in fielded 
equipment and should share this information with the design team. Answers such as “the 
LM understands the importance of reliability” and “they could provide better feedback at 
the part level, from the field and depots” demonstrate that there is a need for this 


information. 
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As far as the usefulness of the input from the LM is concerned, there was a mixed 
reaction. There typically was no answer given for these two questions because of the 
lack of contribution from the LM. On the rare occasions that there was input from the 
LM, some thought it useful and some did not. The answers indicated that education of 
the logistics personnel is a key issue. One person responded that “the logistics group 
does not have the expertise required to impact the early design” and that “logistics 
managers appear to be lacking the technical skills needed to impact the early design.” All 
indications are that logistics is currently not a part of the design process, where they 
could have the most impact. 


D. LESSONS LEARNED 


The following lessons can be extracted from the literature review and the 
interview responses. 


1. “Design in” Reliability Using Highly Reliable Parts, Materials, and 
Processes 


Achieving reliable, available and maintainable systems requires a disciplined 
systems engineering approach that starts early on in a program. Reliability can be 
achieved with the help of highly reliable parts, materials, and processes. It is not 
something that can be tested in later; it must be part of the design process while its still 
cost-effective. 


2. Know And Understand Your Part And Material Application 
Requirements 


There needs to be a clear determination of the system performance requirements, 
including reliability. The system requirements will get allocated to the lowest levels in 
the design. The impacts of not establishing a proper requirement will reverberate down 
to the part and material level. 


3. Test At All Levels of the Design; Test Early; Test Often 


The ability to design a reliable system requires early and continuous testing from 
the system level down to the parts and materials that make up the system. This testing is 
necessary in order to verify and monitor successful achievement of all the performance 


requirements, including reliability. 
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4. Know Your Part Suppliers 


Reliability issues are not always at the system or Line Replaceable Unit (LRU) 
level. Details are required all the way down to the part and material level. A 
microcircuit the size of a penny can end up causing a multi-million-dollar weapon system 
to fail. Attention to the underlying manufacturing design and manufacturing processes of 
the contractor, their subcontractors, and vendors are needed to ensure part and material 
reliability. 

5. Know And Understand The Inherent Design Of The Parts 


Predicted performance, even at the part and material level, can be overstated. 
Apply a null hypothesis to these claims—i.e., they are untrue until proven otherwise in 
the form of valid test results with confidence in the numbers. Just because a supplier 
claims something on the data sheet does not mean that the product has been verified at 
those parameters. Use available data if it is applicable to your system, but always have a 
contractor prove his/her reliability claims, even at the part and material level. 


6. Communication is a Key Part of a Successful PM&P Program 


Successful communications will ensure the best possible outcome of any plan or 
program, and this is no different fora PM&P Program. This will require communication 
and coordination among customers, design engineers, logisticians, contractors, etc. A 
program with an atmosphere of open dialogue and communication is well on its way to 
achieving its long-term goals. 


E. SUMMARY 


The analysis in this chapter provides a valuable insight into the issues that affect 
the use of highly reliable parts, materials and processes in weapons systems. The chapter 
provided useful examples and a set of factors that are used to mitigate the risk of using 
unreliable parts and materials. The analysis was structured around five management 
themes and attempted to pinpoint either practices to implement in a program, or common 
pitfalls that PM should avoid. Finally, the chapter outlined lessons learned from the 
interview process and the background data gathered as part of this research. Chapter V 


summarizes this topic, draws conclusions, and makes recommendations. 


SD 
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Vv. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 


This thesis evaluated the current process for managing PM&P Programs on 
weapon systems and identified some of the common challenges, pitfalls, and lessons 
learned. The key issues and challenges were derived from interviews with individuals 
responsible for implementing PM&P Programs within the Navy, Air Force, and Army 


and with their contractors on weapon system programs. 


This chapter offers conclusions and recommendations for managing PM&P 
Programs. It also identifies the correlation between highly reliable parts and materials, as 
well as overall reliability and critical tactics and strategies that PMs and LMs can use to 
maximize the inherent reliabilities of system components. These conclusions and 
recommendations are based upon research analysis and survey responses. This chapter 
then provides brief answers to the primary and secondary research questions and, finally, 
makes recommendations for further study. 


B. CONCLUSIONS 


Based on the literature review and an analysis of interview results, the researcher 


has drawn the following conclusions: 


1. Idealistic Reliability Estimates from Contractors Appear to be 


Systemic in Weapon Programs 


One of the greatest challenges facing a weapon system program can be 
establishing the reliability of the equipment. Program and Logistics Managers must deal 
with unrealistic reliability estimates from contractors, who are under pressure to win and 
maintain contracts. The impact of a flawed figure of merit is twofold. Not only will the 
weapon system be considered a failure during testing, but the development of a 
sustainment package based on a flawed reliability factor also will produce sparing 


shortages, test equipment shortages, and decreased readiness. 
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2 Highly Reliable Parts, Materials, and Processes are a Fundamental 


Part of Highly Reliable Weapon Systems 


Simply stated, highly reliable parts, materials, and processes are the building 
blocks of a highly reliable weapon system. While the system design approach is largely 
responsible for the weapon system's functional performance, reliable parts and materials 
are essential. Therefore, to insure that the reliability performance is being met, one has to 
consider and evaluate part and material requirements, their implementation and 
verification, and the requirements translation to the lowest part, material and process 


applied in the system. 


3. Successful Communication is a Vital Part of Optimizing the Inherent 


Reliability of a Weapon System 


Communication must be emphasized at all levels. Communication is a tool that 
can assist the program office in avoiding design failures. All team members must be 
aware of the costs of disorganization and poor communication. A weapon system 
program must establish an atmosphere of open dialogue and require personnel to 


participate in all assigned IPT meetings. 


4. A PM&P Program Plan is a Critical Element in the Design of a 
Highly Reliable Weapon System 


There must be a clear determination of the system performance requirements, 
including reliability. The system requirements will get allocated to the lowest levels in 
the design. The impacts of not establishing a proper requirement will reverberate down 
to the part and material level. A PM&P Program Plan is critical because it will help a 


program know and understand part and material requirements. 


5. Part and Material Testing is an Essential Part of Weapon System 


Development 


The ability to design a reliable system requires early and continuous testing from 
the system level down to the parts and materials that make up the system. This testing is 


necessary in order to verify and monitor successful achievement of all the performance 
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requirements, including reliability. Programs often fail to perform part and material 


testing due to inadequate upfront reliability testing and a lack of funding. 
6. Continuous Monitoring of Suppliers is Crucial to High Reliability 


Programs need to monitor suppliers for changes to parts, materials, and processes 
that would affect performance. Attention to the underlying manufacturing design and 
manufacturing processes of the contractor, their subcontractors, and vendors is needed to 
ensure part and material reliability 


C. RECOMMENDATIONS 


The following recommendations can be used to maximize the inherent reliability 


performance of weapon systems. 


1. Establish a True (Measured) Reliability Figure of Merit, then 


Support that Number and not Unrealistic Contractor Estimates 


During initial acquisition, contractor engineers provide reliability estimates that 
are critical in establishing sparing levels and technical support criteria. Experience has 
shown that contractors typically use optimistic reliability predictions. [Ref. 17] Weapon 
systems must dedicate the necessary resources to verify the true reliability and then 
support that number. The failure of a weapon system to achieve its projected reliability 
levels will have a severe impact on the operation and support (O&S) costs and 


operational availability (Ao). 


Ze “Design in” Reliability Using Highly Reliable Parts, Materials, and 


Processes 


The use of highly reliable parts, materials, and processes to “design in” inherent 
reliability can be facilitated with the following two techniques. First, weapon system 
programs should establish a Part and Material Review Board (PMRB). The PMRB 
would be responsible for implementing effective parts and materials management and for 
promoting commonality of parts and processes across product lines. The PMRB can 
enhance the implementation of concurrent engineering, by including representatives from 
seven areas: 1) design engineering; 2) procurement; 3) manufacturing; 4) quality; 5) 
subcontractors and suppliers; 6) customer; and 7) logistics. 
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Weapon system programs should also require their contractors to establish a 
FRACAS that requires root-cause corrective action at the part and material level. A 
FRACAS is an important tool in the solution of complex reliability problems. A well- 
defined FRACAS will catch part and material problems early in the life cycle, before 
they cause high-profile failures. 


3: Weapon System Programs Must Promote Logistics Involvement 


Most weapon system programs already use IPTs as a management technique. 
They need to take the next step and involve the logistics manager early in the system life 
cycle. Most weapon system programs do not have the Logistics Manager involved with 
the PM&P Program. However, weapon system programs should make sure that a 


logistician is a permanent member on any part- and material-related IPT. 
4. Know and Understand Part and Material Application Requirements 


As early as possible in the system life cycle, require all contractors and 
subcontractors to develop a comprehensive PM&P Plan based on MIL-HDBK-512 or 
American Institute of Aeronautics and Astronautics (AIAA) industry standard document 
AIAA R-100. This PM&P Plan should: 1) explicitly define a PM&P Program that will 
ensure the use of parts that meet contractual requirements; 2) reduce proliferation of parts 


through standardization; and 3) enhance equipment reliability and supportability. 
5. Require and Support Part and Material Testing 


Weapon system programs must recognize that they should perform qualification 
testing at the part and material level. Funds should be allocated for independent testing 
of piece parts and materials. The component engineers must know that there are no 
barriers to testing of parts where they feel it is required. The PM&P Program IPT should 
establish design review criteria and make them a standard part of the design review 


process. 


Weapon system programs also must plan for PM&P Program activities. Funding 
is often not allocated to the PM&P Program or is later withdrawn because of competing 
priorities. A long-term approach will help ensure that the PM&P Program has adequate 
funding and support. 
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6. Continuously Monitor Suppliers 


A weapon system program must ensure that contract requirements flow down to 
subcontractors, suppliers, and distributors. PM&P Program personnel must participate in 
the technical evaluation of suppliers and in the review and approval of suppliers’ 
manufacturing processes and parts changes. DoD should re-evaluate the policy of strict 
adherence to performance-based requirements and require contractors to participate in 
programs like GIDEP. Since GIDEP's inception, participants have reported more than 
$1 BILLION in prevention of unplanned expenditures. This means that, without 
GIDEP, participants could have potentially realized additional expenses exceeding $1 
billion. [Ref. 30] 

D. ANSWERS TO RESEARCH QUESTIONS 


Primary Research Question 


How is the selection of highly reliable parts and materials an essential part of the 
overall reliability development process, and what critical tactics and strategies can 
Program and Logistics Managers use to maximize the inherent reliabilities of system 


components? 


Achieving reliable, available and maintainable systems requires a disciplined 
systems engineering approach that starts early on in a program. Maximizing system 
reliability is something that can be achieved with the help of highly reliable parts, 
materials, and processes. Reliability cannot be tested in later, but must be part of the 
design process while it is still cost effective. A microcircuit the size of a penny can end 
up causing an entire multimillion-dollar weapon system to fail. The following are 
recommended steps a Program Manager should consider taking in an effort to optimize 


weapon system inherent reliability: 


Provide adequate funding for a PM&P Program that includes testing at the part 


and material level. 


Develop a comprehensive PM&P Program as early as possible in the system life 


cycle. 
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Ensure that adequate parts, materials, and process requirements are placed in 


contracts. 
Involve PM&P experts early in the process. 
Implement a FRACAS down to the part and material level. 
Monitor and emphasize communication with suppliers. 
Utilize IPTs, including a PM&P IPT. 


The following are recommended steps a Logistics Manager should consider 


taking in an effort to optimize weapon system inherent reliability: 


Make sure that the design IPTs understand the logistics impacts of the early 


design. 


Make sure that the PM understands the logistics impacts and importance of 


reliability. 
Provide feedback at the part and material level, from the field and depots. 


Provide education for your people. Most logisticians lack the technical skills 


needed to impact the early design. 


Subsidiary Research Questions 


The following subsidiary questions focused the author’s efforts in answering in 


the primary research question. 


a. What technology, tools, and techniques are available to ensure the proper 


selection of reliable parts and materials? 


Many tools and techniques can be used to insure the proper selection of reliable 
parts and materials. Listed below are the most frequently utilized design tools to ensure 


that reliability is “built in” early in the design of a weapons system: 
Reliability Prediction Analysis 


Circuit Tolerance Analysis 
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Sneak Circuit Analysis 

Physics of Failure Tools 

Part Derating 

Government Industry Data Exchange Program 

b. How has acquisition reform changed the way a PM&P Program is managed? 


Acquisition reform has significantly increased the risks to PM&P Programs. Prior 
to acquisition reform, a PM&P Program was strictly controlled by the Government 
through contractually required military specifications and standards. After acquisition 
reform, military specifications and standards could no longer be used. There was no 
longer one uniform process; there was the loss of lessons learned intrinsic to the military 
specifications and standards, and all requirements had to be specified in performance- 
based terminology. This has placed a premium on the ability to communicate. 
Concurrent engineering and the use of IPTs are powerful management techniques that 


can reduce some of the risks that were created. 


c To what extent does commercial industry differ in its approach to part and 
material selection, and can the military leverage these best practices to improve 


performance in military systems? 


The DoD already leverages off commercial practices, where practical, in its part 
and material selection. It also is apparent that commercial industry has not stepped up 
and developed “best practices” or initiatives to replace part and material military 
standards and specifications. Where part and material requirements and testing are 
concerned, commercial industry leverages off the DoD. Most of the differences between 
the military and commercial industry can be attributed to a different business 
environment. There are also basic design differences between parts and materials used 
for military vs. commercial applications. Most military designs are used in very harsh 
environments, whereas most commercial components are designed for a protected 
environment. Commercial parts and materials many times are not subjected to shock, 


vibration, pressure or moisture and have an application life span of less than five years. 
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The military environment is much more severe and requires parts and materials with an 


application life span of ten to 20 years. [Ref. 31] 


d. How can Program and Logistics Managers mitigate part and material 


risks? 


The most important thing a PM can do to mitigate part and material risks is to 
make maximum use of IPTs. IPTs will increase the flow of communication and promote 
the involvement of parts and materials engineers at the earlier stages of design. The most 
important thing an LM can do to mitigate part and material risks is to get involved early 
in the IPT process. The LMs also need to develop the technical expertise to influence the 
design of the product. 

E. RECOMMENDATIONS FOR FURTHER STUDY 


The following are recommended topics for additional research: 

1. Evaluate the kind of data currently collected in the field and determine how to 
best provide feedback to a PM&P Program so there can be improvement. 

2. Analyze the best methods and approaches for incentivizing reliability, not 
only in contracts, but also in project offices. 

3. Perform a more complete study on how commercial industry manages parts, 
materials, and processes. Assess if anything can be adapted for weapon 
system development. 


F. THESIS SUMMARY 


Managing reliability performance on a weapon system demands _ the 
implementation of effective management strategies that balance cost, schedule and 
performance against reliability risks over the course of a weapon system’s development 
and fielding. Although reliability is composed of many parts, one of the keys resides in 
early identification of upfront cost-effective opportunities for improving reliability 
performance, as well as mitigation of associated risks during design. Predictable and 


reliable performance in the field is the desired end state. 


Highly reliable parts, materials, and processes are the building blocks of a highly 


reliable weapon system. While the system design approach is largely responsible for the 
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weapon system's functional performance, the PM&P Program approach is responsible for 
a large part of the weapon system's reliability performance. Therefore, to insure that the 
reliability performance is being met, one has to consider and evaluate the PM&P Program 
requirements, its implementation and verification, and the requirements translation to the 
lowest part, material and process applied in the system. The goal of a PM&P Program is 
the selection of parts, materials, and processes that can withstand the manufacturing 
environment and reliably perform the needed function for the design life of the product in 


the intended environment. 
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